Repository navigation
Conversation
Add EID value types, an additive KV schema for provider-written EIDs with owner and expiry, EID-level fields on the OpenRTB Eid, and a resolver that serves only live UIDs to auctions, x-ts-eids, identify and the admin auction preview. Auction EIDs merge per (source, inserter, matcher, mm, ext) with first-occurrence UID dedupe. KV writes over 16 KiB are refused. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Top-level document navigations on the publisher route receive a single-use write permit; finalize plans one row change (browser ingest, provider results, prune, size cap) and commits it once without retry. Non-navigation requests and redirects never write, integration proxies no longer mint ECs, and new ECs are created by the page-load write instead of at generation. Withdrawal tombstones only the active EC and keeps the EC cookie when the tombstone does not land; the withdrawal completion marker is the one deliberate extra write. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add the IdentityProvider/ProviderKind interface and [[ec.identity_providers]] config, response validation, a deadline-bounded LookupTransport over select_until, the IdentityCoordinator with consent, allowlist and budget checks, the trusted-server-identity-providers crate (no kinds yet), and Fastly wiring that dispatches before the origin fetch and collects before the single page-load write. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Orphan recovery minted a new EC ID and then discarded the navigation's provider results because they were looked up under the orphaned ID. Lookups depend on the browser, not the EC ID, and the orphan has no row, so retarget them to the replacement as the spec requires. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Collection that starts after its deadline takes one nonblocking snapshot. A request still in the headers state moved to the body state without reading bytes, which counted as no progress, so a response that had fully arrived was abandoned unread and reported as a timeout. Count entering the body state as progress. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A response body the host already holds can report not ready until the host services async I/O once, so an expired snapshot abandoned identity responses that arrived well before the deadline and reported them as timeouts. Give each pending body one host select capped at 1 ms before abandoning it. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…not fill Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Page-load writes only run on real navigations, which browsers mark with both sec-fetch-dest: document and sec-fetch-mode: navigate. Without the mode header the consent withdrawal scenario never got a write permit, so the EC cookie was kept. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A lookup sends the EC ID to the provider before finalize can decline the EC, so GPC, sale opt-out and unknown-jurisdiction requests must not dispatch. Matches the gate pull sync already applies. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
… read LiveRamp envelopes are about 700 bytes, so the 512-byte limit dropped every one from KV and the auction. Rows written under a higher limit now load with the oversized IDs removed instead of failing as a whole. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
304f395 to
2de6d0d
Compare
|
@dhruv8sh, directionally the layout here is not compatible with V2 (the 55 pull requests tracked in #777, merged in order). This pull request creates one shared The V2 spec is identifier-modules and pluggable-providers, with the folder layout in integrations-crate, all in #1342. The associated pull requests are #1382 (a module hands core its identifiers, starting with the Prebid EID cookie) and #1343 (the Edge Cookie as a module, with the HMAC module built in). Could you evaluate the V2 spec and those pull requests before progressing this one? |
…readable rows Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…or gate Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
#1487's head 64674f9 stacks on #1526's b53c6e2 and merged #1557's 76e1dd5; each merge resolved its own conflicts, so this merge needs no resolution. It brings server-side identity provider collection with one page-load write (#1567), with lookups on EdgeZero's outbound client and the KV code on #1526's async store.
Brings PR #1488's vendor identity providers onto PR #1487's new head, which carries PR #1487 onto PR #1526 (EdgeZero cutover, local partner API answers, HTTP auction opt-in and awaitable EC persistence). How each area was resolved: - Coordinator: PR #1488's cookie reads, GDPR source exclusion, opt-out memory and merged results run inside PR #1487's batch-shaped dispatch. The lookups a navigation still needs are built first and start together in one EdgeZero batch; `PendingLookups::is_empty` counts started requests along with cookie, generated and early results. - Fastly entry point: `main.rs` takes PR #1487's EdgeZero version. The provider cookies PR #1488 hands to after-response lookups go into the after-response `LookupContext` in post-delivery work; the request-side cookie capture merged unchanged. - Async KV: PR #1488's new write-plan and identify tests await the store. - Providers crate, CLI and settings: merged without conflict; providers stay pure and make no HTTP calls of their own. - Cargo: the textual merge pointed `crypto-common` at a `getrandom` the lockfile no longer held; cargo resolved it to the locked 0.4.2. The rest of the lockfile delta is PR #1488's own.
Brings PR #1489's pull-sync provider and EID mirror onto PR #1488's new head, which carries PR #1487 onto PR #1526 (EdgeZero cutover, local partner API answers, HTTP auction opt-in and awaitable EC persistence). How each area was resolved: - Pull sync: core `ec/pull_sync.rs` stays deleted, so the base's EdgeZero group sender and its tests go with it. Pull sync runs as PR #1489's `pull-sync` after-response provider, sent through the post-delivery client in the lookup batch. Post-delivery work is now telemetry, then after-response lookups; the separate pull-sync step, its graph factory and the unused clock are removed, and PR #1489's "nothing due" check runs before the live row read. `require_identity_graph`, now used only by tests, is test-only. - Coordinator: PR #1489's dispatch order and per-navigation pull-sync cap run inside PR #1487's batch-shaped dispatch; the cap counts each queued pull-sync request. - Fastly app: `EcFinalizeState` drops both the old `services` and `is_real_browser`; the bot gate moves into after-response capture as in PR #1489, with the base's request services and async batch sync. - Async KV and finalize: PR #1489's removals in `kv.rs` (snapshot partner upsert) and the pull-sync marker stand; the mirror and the inner finalize are awaited, and PR #1489's new tests await the store. - Publisher: keeps the base's EdgeZero origin request without the marker preload inputs PR #1489 removed. - Providers crate: PR #1489's pull-sync providers and PR #1488's SharedID cookie defaulting both apply when the phase coordinators are built. - Docs and CHANGELOG: the base's auction wording with PR #1489's identity bullets; the onboarding step names after-response lookups.
…ntity #1489's head a7262c7 stacks on #1488 (8183c3c), which stacks on #1487 (64674f9); each merged the one below it and resolved its own conflicts, so this merge needs no resolution. It completes the provider stack (#1567): the vendor identity providers and cookie hooks (#1488), and the bounded identity mirror with pull sync as an after-response provider on EdgeZero's outbound client (#1489).
…ity plan Both integration branches now carry every in-review issue's non-draft pull request. DEC-20 records the new rule that a conflict is fixed in the pull request that lands second, and DEC-21 the merge order the user chose: the Trusted Server pull requests stack linearly on #1506 (stack 1621), so T1, #1525 and #1526 land after the cutover. The spec's Integration branches section and the plan's State section list the new heads, and the wave rows for T1, #1525, #1526, #1506 and #1487 follow the new order.
Core identity app. IdentityApp now carries the provider stack's identity state as well as its own: the partner registry, the page and after-response provider coordinators (#1487, #1489), the EC record store and the rate limiter. IdentityPlatform takes an optional IdentityProviders (the registry the coordinators were built against plus both coordinators); without one the app builds the registry with PartnerRegistry::from_ec and runs no provider, matching the base's non-Fastly adapters. The admin EC lookup awaits the async handler from #1526. Send and Sync. #1526 keeps EcKvStore free of Send and Sync bounds and tests a non-Send store, so the T21 supertrait bounds on EcKvStore and RateLimiter are dropped. The identity app instead holds its record store and rate limiter as Send and Sync trait objects, and IdentityApp::identity_graph hands out a graph over that one shared store through the new KvIdentityGraph::from_shared. IdentityPlatform's identity_graph field becomes record_store accordingly. Fastly app. AppState drops the base's partner_registry, page_identity and after_response_identity fields for the identity app. build_identity_app builds the kind-aware registry and both coordinators from the identity-providers crate, then the app with Fastly's record store and rate limiter. Page lookups and after-response preparation read their coordinators and rate limiter from the identity app; identify, batch sync, auction, page bids and the publisher fallback use its registry and graph. The admin lookups stay mounted by the identity app, so the base's async admin early dispatch is removed. Fastly entry point. finalize_response keeps the base's single block_on around the async EC finalize, which collects the page lookups first. FinalizeInputs keeps the base's AppState, and FinalizeState carries the identity app and the after-response lookups. maybe_identity_graph is gone; the test-only require_identity_graph reads the identity app's graph. Post-delivery. The after-response providers, pull sync included, take the partner registry, graph and rate limiter from the identity app carried in FinalizeState. The base's pull-sync tests build their state through the identity app. Axum, Cloudflare and Spin. The identity app mounts the admin lookups; the base's provider rejection at startup and #1614's local 501 and 405 partner API answers are kept. Tests. The core identity app tests use record stores and the async graph, and two tests cover the carried registry and coordinators. The Fastly identity fixture builds an identity app around its echo-provider coordinator. The adapter identity guard replaces dispatch_pull_sync, which #1489 deleted, with commit_after_response_lookups and also guards PartnerRegistry::from_ec.
A provider that reports an opt-out now yields LookupStatus::OptedOut, while Skipped(OptedOut) only means the provider was skipped because of an opt-out remembered in the row. Only a reported opt-out starts the 30-day memory period. A remembered skip whose opt-out lapsed between dispatch and the write is discarded, so it neither renews the opt-out nor clears UIDs a newer lookup stored. The concurrent opt-out guard still keeps found UIDs from being restored over a newer opt-out. Logs report a provider-reported opt-out as opted_out; skipped_opted_out keeps its meaning. The spec's outcome list and opt-out rules are updated to match. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
The spec placed stored row types in eid_types, merge and the mirror in the coordinator, a mirror method on IdentityCoordinator, a clock field on OutboundLookupTransport and provider cookie reads in PR A. Rustdoc still described finalization classifying new identities, the coordinator returning a row change and withdrawal as the only extra write. Correct each to match the code. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Brings #1489's provider-based pull sync and vendor providers, with #1487 and #1488 at their current heads, into the certificate-check statement for every outbound HTTPS request. The merge is clean. #1489's new commits leave post_delivery.rs as it was, so the PR's after-response pull-sync certificate-check test and its helpers run unchanged, and the base's settings and configuration guide edits sit apart from the PR's certificate-check wording. Cargo.lock is unchanged.
Brings the certificate-check statement for every outbound HTTPS request (#1616), on #1489's current head with #1487 and #1488, into the cross-adapter health and startup-failure contract. The merge is clean. #1489's new commits add provider cookie opt-out handling to the Fastly app's route dispatch, apart from the PR's probe and startup-failure changes, and #1616 touches neither, so every adapter keeps the PR's contract as merged before. Cargo.lock is unchanged.
#1615's head fb25de5 carries the whole chain from #1506 through #1489 at 8623c4e, including the EdgeZero repin to rc/identity 9d02d7b3 and dhruv8sh's latest #1487, #1488 and #1489 commits. It also carries prk-Jr's line #1489 <- #1616 <- #1617 <- #1615: certificate checks on every outbound HTTPS request (#1556), the cross-adapter health and startup-failure contract (#1142) and the creative proxy identity disclosure policy (#1233). Each PR merged the one below it, so this merge is clean.
|
Checked |
Summary
atype,ext) and serves them unchanged into OpenRTB and Prebid, so bidders see the same shape the provider returned.This PR registers no provider kinds yet, so none of the provider paths run in production until the vendor PR lands. Follow-up PRs stack on this branch: mirror/pull sync (#1489), then the per-vendor provider implementations (#1488).
Changes
ec/eid_types.rs,ec/kv_types.rs,ec/kv.rs,openrtb.rs,ec/prebid_eids.rsec/write_plan.rs,ec/page_load.rs,ec/finalize.rs,publisher.rsec/provider.rs,ec/coordinator/*,ec/registry.rs,settings.rs,config.rs, newtrusted-server-identity-providerscrate, adapterapp.rs/main.rsec/coordinator/after_response.rs, adaptermain.rsdocs/guide/configuration.md,docs/guide/edge-cookies.md,docs/guide/integrations/prebid.md,trusted-server.example.tomlec.identity_providersconfiguration docs and exampledocs/superpowers/specs/2026-10-30-s2s-eid-collection-design.mdCollection
collect = "page"(default) runs before the response withinidentity_budget_ms(default 200).collect = "after_response"runs after the response is sent withinidentity_after_response_budget_ms(default 2000).tcf_vendor_id, when set, is checked only where GDPR applies or a TCF string is present. A KV read failure skips every provider. When a provider's vendor consent is denied, its stored IDs are also withheld from bidders,x-ts-eidsand/identify.opt_outsfield on the row, and the provider is not called again until it lapses. Older builds ignore the field.rate_limit_per_houris per EC hash, so clearing cookies does not reset it. On Fastly it is enforced per minute with a minimum of one, so values below 60 behave like 60.Storage and writes
ts-eidscookie, so an opted-out ID cannot come back from the browser.main) and expires the cookie once every tombstone is in place.row_fulland continues.Behaviour changes from
mainSec-Fetch-Dest: documentandSec-Fetch-Mode: navigate, the same check that permits its row write. Clients that send no Fetch Metadata no longer get a new EC./auctionor page-bids request body go to bidders but are never persisted.user.idand no provider lookup runs for it until a later navigation finds its row.Cleanup
ProviderResponsedebug output hides the body and any request context.Closes
Part of #1567 and #1245; #1489 closes #1567. Two spec requirements this PR leaves open are delivered higher in the stack by #1642 (#1591): ID-8, saving
/auctionbody EIDs, and SH-2, removing the per-requestthread_local.Spec
Spec:
docs/superpowers/specs/2026-10-30-s2s-eid-collection-design.md, included in this PR.Test plan
cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spinclippy-fastly,-axum,-cloudflare,-cloudflare-wasm,-spin-native,-spin-wasm,-cli,-codegen)cargo fmt --all -- --checkscripts/lint-wasm-clock.shcd crates/trusted-server-js/lib && npx vitest run(plusnpm run formatandnpm run lint)./scripts/test-cli.sh, domain lint againstorigin/main, CI Node pin guard, docs-proposal scriptsfastly compute serveChecklist
unwrap()in production code — useexpect("should ...")logmacros (notprintln!)